Sufficient Dimensionality Reduction with Irrelevant Statistics

The problem of unsupervised dimensionality reduction of stochastic variables while pre­ serving their most relevant characteristics is fundamental for the analysis of complex data. Unfortunately, this problem is ill defined since natural datasets inherently contain al­ ternative underlying structures. In this paper we address this problem by extending the re­ cently introduced "Sufficient Dimensionality Reduction" feature extraction method [7], to use "side information" about irrelevant struc­ tures in the data. The use of such irrelevance information was recently successfully demon­ strated in the context of clustering via the Information Bottleneck method [1]. Here we use this side-information framework to iden­ tify continuous features whose measurements are maximally informative for the main data set, but carry as little information as possi­ ble on the irrelevance data set. In statistical terms this can be understood as extracting statistics which are maximally sufficient for the main dataset, while simultaneously max­ imally ancillary for the irrelevance dataset. We formulate this problem as a tradeoff op­ timization problem and describe its analytic and algorithmic solutions. Our method is demonstrated on a synthetic example and on a real world application of face images, show­ ing its superiority over other methods such as Oriented Principal Component Analysis.